In a complex system, such as an aircraft, monitors are typically used to ensure that the system is operating as expected. When a monitor detects an undesirable result, the monitor can trigger a message or other event. Results of monitors may also be used to change an operating condition of the system, such as disabling a component or subsystem and switching to a backup component or subsystem. For monitors to be effective, they must trip as expected to avoid exceeding potential hazard conditions, and only trip when required to avoid nuisance alerts. When a monitor trips but no fault is found upon further inspection, maintenance overhead is increased as the monitored system is analyzed in detail to attempt to locate a root cause of a non-existent issue.
In some instances, monitor limits used to trip monitors are selected by domain experts based on past experience. In other instances, monitor limits represent a fixed percentage of deviation relative to an expected value. A further alternative is to select a midpoint between a peak operating level and a hazard level. While a number of monitor limit selection techniques can be effective, they may not fully account for potential variations upon system integration. For example, sensor relocation, cable length changes, environmental conditions, transients, aging effects, manufacturing variations, accuracies, and cascaded components can impact actual monitor performance versus expected monitor performance. Selecting an unreasonably short confirmation time or an unreasonably tight tolerance on monitor limits can lead to nuisance alerts. Additionally, leaving monitors active when the monitored component or subsystem is not in an operative state can lead to nuisance alerts.
What is needed is an improved system and process to measure robustness and failure detection capabilities of monitors in a system, thereby quantitatively validating monitor performance or initiating a change process.